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ABSTRACT 

This work was performed under a task entitled "Development 
and Assessment of ADL Prototypes." This task is intended to promote 
collaboration by the Services and by other government and academic partners 
in developing technology-based instruction. It is an essential component of 
the Advanced Distributed Learning (ADL) initiative being undertaken by the 
Department of Defense (DoD) in cooperation and coordination with the Office 
of Science and Technology Policy (OSTP) . This particular document is intended 
to provide an overview of technology-based instruction. Writing and printed 
books have influenced learning profoundly. It seems reasonable to view the 
emergence of writing as the first major revolution in learning. By making the 
content of learning, teaching, and educational material widely and 
inexpensively available anytime and anywhere, the development of books 
printed from movable type effected a second revolution in learning. Can we 
improve on books? Computer technology is a possibility because computers can 
adapt the content, sequence, type, difficulty, granularity, and so forth of 
their presentations to learners or problem solvers based on ongoing, dynamic 
assessments of individual needs. For this reason, computer technology may be 
effecting a third revolution in learning. While preserving the capabilities 
of books to present the content of instruction anytime, anywhere, computers 
can also provide the interactions of teachers, instructors, tutors, and 
mentors as needed by individual learners and users. This capability may 
produce a third revolution in learning, with a significance equal to that of 
the development of writing and books. This document addresses this "Third 
Revolution" in learning by assessing the impact of computer-based technology 
on learning. Consequences of this third revolution include making 
individualized tutoring widely accessible and affordable; permitting 
interactive, individualized learning to take place anytime, anywhere; 
changing the way that educational institutions and educators view learning by 
producing radical changes in the practice and processes of instruction; and 
bringing into being a nation of life-long learners who are prepared to meet 
future technological challenges and thrive in the global marketplace. 

Includes a glossary of abbreviations. (Contains 43 references.) (Author) 
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PREFACE 



This work was performed under a task entitled “Development and Assessment of 
ADL Prototypes.” This task is intended to promote collaboration by the Services and by 
other government and academic partners in developing technology-based instruction. It is an 
essential component of the Advanced Distributed Learning (ADL) initiative being under- 
taken by the Department of Defense (DoD) in cooperation and coordination with the Office 
of Science and Technology Policy (OSTP). This particular document is intended to provide 
an overview of technology-based instruction. 
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EXECUTIVE SUMMARY 



Writing and printed books have influenced learning profoundly. The emergence of 
writing brought about a revolution in learning by freeing learners from requirements for 
face-to-face interaction with sages and other human sources of knowledge, lore, and 
information. Although learning became neither inexpensive nor universally available after 
the development of written language, its content became available anytime, anywhere. 

A second revolution occurred with the development of moveable type and books. 
These developments also made the content of learning, teaching, and educational materials 
available anytime, anywhere — but more widely and, significantly, less expensively. 

Can we improve on books? Computer technology arises as a possibility. Computers 
can adapt the sequence and type of operations they perform to the conditions of the 
moment. In instruction, they can adapt the content, sequence, type, difficulty, granularity, and 
so forth of their presentations to learners or problem solvers based on ongoing, dynamic 
assessments of their individual needs. 

For this reason, computer technology may be effecting a third revolution in learning. 
While preserving the capabilities of books to present the content of instruction anytime, 
anywhere, computers can also provide the interactions of teachers, instructors, tutors, and 
mentors as needed by individual learners and users. This is not something books, movies, 
television, or videotape technologies can do affordably or to any significant degree. By 
presenting the content and the interactions of instruction, computers promise a new and 
significant capability for learning that is available anytime, anywhere. This capability may 
produce a third revolution in learning, with a significance equal to that of the development of 
writing and books. 

Some findings that suggest this revolution is likely and within reach are: 

• Individualization. Experimental studies comparing human tutors using only 
rudimentary techniques of individualization with classroom instruction have 
found effect sizes as large as 2.00 standard deviations favoring the tutored stu- 
dents. Technology makes such tutoring capabilities affordable. 

• Adaptation and interactivity. As a form of individualization, branched 
instruction that adjusts its content and pace for individual learners has been 



compared with a strictly linear presentation of the same instructional material 
using identical delivery devices. Studies found effect sizes of 0.72 and 2.16 
standard deviations favoring the branched approaches and indicate the value of 
adaptation and interactivity in instruction. 

Intensity of instruction. An individual student in classroom instruction is 
required to answer an average of about 0.11 questions an hour. In tutored 
instruction, the number of questions an individual student is required to answer 
has been found to range from 117 to 146 questions an hour. Results from 
some technology-based instruction have been found to average about 
120 questions during 12-minute sessions. The intensity of individual tutoring 
and technology-based instruction far exceeds that of classroom instruction. 

Pace of instruction. The ratio of time the fastest students in a typical class- 
room need to achieve mastery of material being presented compared with the 
time needed by the slowest students is about 1:5. This spread of capabilities is 
very difficult for human classroom teachers to satisfy, despite their heroic 
efforts to do so. Some students will inevitably be bored, and others will be lost. 
Many studies have found that allowing students to use technology-based 
instruction to progress at their own rates and receive material tailored to their 
own needs reduces overall time-to-mastery by about 30 percent. 

Cost-effectiveness. When compared with reduced class size, increased 
instructional time, peer tutoring, and professional tutoring, technology-based 
instruction was found to be the most cost-effective approach among these 
alternatives. 

Intelligent tutoring systems. Intelligent tutoring systems intended to replicate 
the practices of expert human tutors have demonstrated effect sizes of 0.84 to 
1.00 standard deviations when compared with classroom instruction. These 
systems appear to have raised significantly the ceiling for instructional 
improvements potentially available from technology-based instruction. They 
may end up using approaches that are different and far more powerful than 
those used by human tutors. Through the use of web-accessed instructional 
objects and personal (perhaps wearable) learning associates, these systems may 
yield unexpected and unanticipated results. 

seems reasonable to conclude that this third revolution will: 

Make the functionalities of individualized tutoring widely accessible and 
affordable 

Permit interactive, individualized learning to take place anytime, anywhere 

Eventually bring about profound changes in our educational institutions and in 
the roles and responsibilities of the people (instructors, students, and adminis- 
trators) who work in them 



Help bring into being a nation of life-long learners who are well prepared to 
meet the challenges of technological change and thrive in the global market- 
place 

Produce radical change in the practice and processes of instruction 
Change what we do and what we aspire to do in instruction. 



TECHNOLOGY, THE COLUMBUS EFFECT, AND 
THE THIRD REVOLUTION IN LEARNING 



A. INTRODUCTION 

The apocryphal tale goes something like this: 

An Enlightened Leadership discovers that relatively minor investments in 
educational technology can significantly enhance the capabilities — the pro- 
ductivity, competitiveness, and competence — of their domain. The next step, 
of course, is to appoint a “Blue-Ribbon” Committee. The task of the 
Committee is to design the ideal technology for education. The Committee 
meets, deliberates, and issues specifications for the new technology. 

Physically, it must be rugged, lightweight, and easily portable — available 
anytime, anywhere. It must operate indoors and out, under a wide range of 
temperature, humidity, and other environmental conditions, and it must 
require only minimal, if any, external power support. Functionally, it must 
provide easy, rapid, and random access to high-quality text, black and white 
or full-color graphics, and high-resolution photographs. It must include an 
interface that is easily understood and usable by all, preferably communi- 
cated in natural language. It should allow self-pacing (i.e., learners should be 
able to proceed through instructional content as rapidly or as slowly as 
needed). It should be suitable for life-long learning and readily available to a 
wide range of users in home, school, and workplace settings. Economically, 
it must be inexpensive, or as the Committee reports “require only minimal 
financial investment on the part of potential end users.” 

The Enlightened Leadership receives the Committee’s report with relief. 
Development of the technology will require no lengthy research and devel- 
opment, no new taxes, no new infrastructure, and no difficult political or 
administrative decisions or compromises. In fact, the only thing it will 
require is business as usual. The reason may be as obvious to the readers as 
it is to the Enlightened Leadership. The recommended technology is, of 
course, the technology of books — which are already available and in place. 

Writing and books effected revolutions in learning or, more precisely, how we go 
about the business of learning. Before their appearance, during the earlier 100,000 or so 
years of human (i.e., homo sapiens) existence, instruction took place person to person. It 
was expensive and slow and produced uneven results because it depended heavily on the 
teacher’s knowledge, capabilities, and instructional expertise. Matters concerning more than 
basic subsistence reached very few people. 



Writing developed about 7,000 years ago and progressed from picture-based ideo- 
graphs to consonants and vowels represented with alphabet-based phoneticization by per- 
haps 1000 BC. It allowed the content of advanced ideas and teaching to transcend time and 
place. Because of that capability, it effected a major revolution in learning. People with 
enough time and resources could study the words of the sages who went before them with- 
out having to rely on face-to-face interaction or the vagaries of human memory. 

As discussed by Kilgour (1998), books (i.e., something beyond mud or stone tab- 
lets) were copied first on papyrus and later on parchment rolls until about 300 BC when the 
Romans began to sew sheets of parchment together into codices. These codices resembled 
the books of today and allowed easier and random access to the content. They were also 
less costly to produce than papyrus rolls because they were based on locally available 
parchment made from animal skin and allowed content to be placed on both sides of the 
sheets. Use of paper prepared from linen and cotton [about 100 AD (China) and 1200 AD 
(Europe)] made books even less expensive. Still, books were by no means inexpensive. 
However, their lowered costs made them more available to a literate and growing middle 
class who increased the demand for books and the learning they provided. This increased 
demand, in turn, lowered costs even further. 

The full technology called for by the Committee mentioned previously finally 
became available with the introduction of books printed from moveable type (Kilgour, 
1998). These printed books were first produced in China around 1000 AD and in Europe in 
the mid 1400s. At this point, the content of knowledge and teaching became widely and 
increasingly inexpensively available. The only item lacking from the Committee’s list of 
specifications was the availability of high-resolution photographs, which had to await the 
development of photography in the mid 1800s. 

Writing and printed books have impacted learning profoundly. It seems reasonable 
to view the emergence of writing as (among other things) the first major revolution in 
learning. Although learning — the acquisition of knowledge — was still neither inexpensive 
nor widely available, it no longer required face-to-face interaction with sages. By making the 
content of learning, teaching, and educational material widely and inexpensively available 
anytime and anywhere, the development of books printed from moveable type effected a 
second revolution in learning. 
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THE THIRD REVOLUTION IN LEARNING 



At this point, we might consider an argument often advanced but best articulated by 
Clark (1983). In discussing all the means (media) we have for delivering instruction, Clark 
asserted, “the best current evidence is that media are mere vehicles that deliver instruction 
but do not influence student achievement any more than the truck that delivers our groceries 
causes changes in our nutrition” (p. 445). Of course, Clark had in mind more recently 
developed media than books, but printed books seem fair game for discussion in this 
context. 

Are books “mere vehicles” for delivering instruction? Do they deserve more credit 
than that for influencing student achievement? Just as trucks are essential components in an 
infrastructure that has improved the nutrition of nations, books are essential components in 
an infrastructure that has improved the learning, performance, and competence of people 
everywhere. Both trucks and books may be vehicles, but their contributions seem to be more 
fundamental than “mere.” 

However, the heart of Clark’s argument remains sound. Books do not guarantee 
learning or student achievement. Ignorance remains plentiful, even though books have 
diminished its supply appreciably. Clark’s concern can be summed up by the notion that 
technology alone does not define an instructional approach. What matters is what is done 
with the technology. This point of view seems fair and unequivocal. The presence of any 
technology does not guarantee that effective instructional content, effective ways to present 
this content, or even the unique strengths of the technology itself will be present or used. On 
the other hand, the absence of a technology is a reasonable guarantee that its functionalities 
will be missing. Without printed books, we may be back to the 1400s. 

How do we improve on books? Do we have anything better? Computer technology 
arises as a possibility. One of the most important statements in higher order computer lan- 
guages (based necessarily on what is available in every digital computer’s lower-order 
instruction set), is the “if’ statement. This statement is of the (very) general form: 

If <some condition> is true, then do <something> ; otherwise, do <something else>. 

We can marvel at the capabilities of computers to perform millions of operations a 
second with perfect accuracy on the immense amounts of data that they retrieve. However, 
the central point for this discussion is the capability of computers to adapt the sequence and 
the type of operations they perform based on conditions of the moment. More specifically, 
they can adapt the content, sequence, type, difficulty, granularity, and so forth of 
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resentations to learners and other users based on their assessment of learners’ and users’ 
current needs. 

For this reason, computer technology may effect a third revolution in learning. 
While preserving the capabilities of writing and books to present the content of excellent 
instruction anytime, anywhere, computers can also provide the interactions of excellent 
teachers, instructors, tutors, and mentors as needed by individual learners. This is not 
something books, movies, television, or videotape technologies can do affordably or to any 
appreciable degree. This interactivity is a new and significant capability. It is the core of 
what future commentators may view as the third revolution in learning. 

C. DOES INTERACTIVITY MATTER? 

Much of the remainder of this discussion centers on whether the instructional inter- 
activity provided by technology matters. Can we expect a revolution in instruction equivalent 
to that wrought by writing and books? What can we say about the nature of this revolution 
and what it implies for instructional practice? 

The question of whether interactivity matters is addressed to some degree by studies 
in which as much of the instruction as possible is held constant except for the level of inter- 
activity. Fowler (1980) and Verano (1987) performed two such studies. Fowler compared 
branched presentations using computer-controlled, adaptive videodisc instruction with 
instruction in which the same materials were held to a fixed-content, linear sequence. She 
reported an effect size of 0.72 (roughly, an improvement from the 50 lh to the 76 lh percentile 
in performance) for ability to operate and locate faults on a movie projector, which was the 
object of her instruction. Similarly, Verano also compared an interactive, adaptive, branching 
approach for presenting instructional material with a strictly linear approach used to present 
identical instruction in beginning Spanish. Both of his treatments used videodisc 
presentations. He reported an effect size of 2.16 (roughly, an improvement from the 50 lh to 
the 98 lh percentile in performance) in end-of-course knowledge. These two studies, among 
others, suggest that interactivity — at least, interactivity as defined by these studies — matters 
and perhaps a great deal. Of course, there is more to the story. 

D. TUTORING AND THE INDIVIDUALIZATION OF INSTRUCTION 

Individualized tutoring (one student working with one instructor) has been viewed 
and used for a long time as an effective instructional procedure. Evidence of its value is 
found in its continued use for instruction in highly complex and high-value activities, such 
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s aircraft piloting, advanced scientific research, and specialized medical practice. Compari- 
sons of one-on-one tutoring with one-on-many classroom instruction might be expected to 
favor individualized tutoring, and they do. However, these comparisons are surprising not 
because of the direction of these findings, but because of the magnitude of the differences in 
instructional effectiveness. 

Benjamin Bloom’s results may be the most widely noted of these. Combining the 
findings of three empirical studies that compared one-on-one tutoring with classroom 
instruction, Bloom (1984) reported a general difference in achievement of two standard 
deviations (roughly, an improvement from the 50 lh to the 98 th percentile in performance) in 
favor of tutoring. On the basis of considerable empirical evidence, these and similar studies 
suggest that differences between the results of one-on-one tutoring and classroom instruc- 
tion are not just likely but very large. 

Why, then, do we not provide these manifest benefits to all our students? The 
answer is straightforward and obvious and has doubtless already occurred to the reader: we 
cannot afford it. The issue is not effectiveness, but cost. Unless our policies toward educa- 
tional funding change dramatically, we cannot afford a single tutor for every student. Bloom 
(1984) popularized this issue as the Two-Sigma 1 Problem. 

In 1975, Scriven argued that individualized instruction was an instructional impera- 
tive and an economic impossibility. Is it? Must instruction remain constrained by this real- 
ity? 

Enter Moore’s (famous) Law. 2 As recounted by Mann (2000), Electronics maga- 
zine interviewed Moore in 1965 and inquired about the future of the microchip industry. To 
make a point, Moore noted that engineers were doubling the number of electronic devices 
(basically transistors) on chips every year. In 1975, Moore revised his prediction to say that 
the doubling would occur every 2 years. If we split the difference and predict that it will 
occur every 18 months, our expectations fit reality quite closely. As Mann points out, the 
consequence of Moore’s Law is that computers that initially sell for $3,000 will cost about 
half that in about 18 months. 

Moore’s Law has implications for our learning applications: Computers are 
becoming exponentially less expensive, and the computational capabilities that we need to 



1 As in two standard deviations. 

2 Gordon Moore is a semiconductor pioneer and co-founder of Intel Corporation. 
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jpport instruction that is very much like individualized tutoring are becoming progressively 
affordable — if they are not already. The issue then becomes this: How we should use this 
increasingly affordable computational power to support learning? 

Since the 1960s (e.g., Suppes, 1964; Atkinson and Fletcher, 1972), we have had 
computer-based instruction (CBI) that could tailor the content, sequence, and difficulty of 
instructional content to the needs of individual learners, and these approaches were shown to 
be effective. The Stanford beginning reading programs presented on Model 33 teletype- 
writers running at 110 baud (about 10 characters per second) with randomly accessible 
digitized audio achieved effect sizes in excess of 0.80 standard deviations (Fletcher and 
Atkinson, 1972). Similar results were obtained for elementary school mathematics (Suppes, 
Fletcher, and Zanotti, 1975). Instructional approaches used in these early programs required 
computers that cost $2-3 million; however, today, computers that cost under $1,000 could 
easily present these same approaches. 

These approaches did not seek to mimic directly the interactions that occur in human 
tutorial instruction. Instead, they attempted to apply results emerging from empirical studies 
of human cognition, memory, and learning as discussed by Suppes (1964) and Atkinson 
(1972). Efforts to provide tutorial dialogue emerged from approaches that were initially 
described as intelligent computer-assisted instruction (CAI) and later as intelligent tutoring 
systems (ITSs) (Sleeman and Brown, 1982; Woolf and Regian, 2000). They have been 
effective, occasionally yielding effect sizes in excess of 1 .00 (e.g., Gott, Kane, and Lesgold, 
1995). These approaches raise the following questions: What accounts for the success of 
one-on-one tutorials? Can we do the same with computers? 

E. INTENSITY OR INTERACTIVITY OF INSTRUCTION 

Graesser and Person (1994) discussed this issue. They compared instruction using 
one-on-one tutoring with classroom practice in two curriculum areas: research methods for 
college undergraduates and algebra for 7th graders. Tutors for the research methods course 
were psychology graduate students, and tutors for the algebra course were high school stu- 
dents. 

Graesser and Person found the following: 

• Average number of questions asked by teachers in a class in a classroom hour: 
3 

• Average number of questions asked by any one student during a classroom 
hour: 0.11 
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• Average number of questions asked by a student and answered by a tutor 
during a tutorial hour: 

- Research methods: 21.1 

- Algebra: 32.2 

• Average number of questions asked by a tutor and answered by a student 
during a tutorial hour: 

- Research methods: 1 17.2 

- Algebra: 146.4. 

These data do not prove hard-core cause and effect, but they show great differences in sheer 
interactivity between two approaches that also show great differences in instructional 
effectiveness. 

Does CBI echo this level of interactivity? Few studies report the number of ques- 
tions students answer per unit of time when using technology. However, this author 
(Fletcher) found that K-3 students who receive technology-based beginning-reading and 
beginning-arithmetic instruction on the previously mentioned 1 10-baud teletypewriters were 
answering 8-12 questions a minute — questions that were individually assigned and whose 
answers were immediately assessed. 

This level of interactivity would extrapolate to 480-720 such questions an hour if 
children of this (or any) age were able to sustain this level of interaction for 60 minutes. 
Instead, these children generally worked with the computer-based materials in daily 
12-minute sessions, which extrapolates to 96-144 individually selected and rapidly assessed 
questions that these children received each day. As mentioned previously, this CAI was 
producing effect sizes in excess of 0.80 standard deviations in comparisons with classroom 
instruction in mathematics and reading. The speed, or pace, with which students were 
allowed to progress through instructional presentations, as much as anything else, may be 
responsible for the success of these and other CAI programs. 

Graesser, Person, and Magliano (1995) point out that the neither the students nor 
the tutors they observed were particularly sophisticated in their use of questions. Specifi- 
cally, they found that the tutorial techniques long advanced by researchers and scholars 3 
were largely absent. About half of the questions asked by the students and their tutors 



3 For example, shaping and fading (Skinner, 1968), scaffolding (Ausubel, 1960; Rogoff, 1990), 
reciprocal instruction (Palincsar and Brown, 1984), error diagnosis and repair (Burton, 1982; van Lehn, 
1990), or advanced motivational approaches (Lepper et al., 1993). 
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required simple yes/no responses. The techniques the tutors used were far from sophisti- 
cated, but these techniques, as the data tell us, were effective. Simple approaches that aim 
primarily to increase interactivity may by themselves fill much of Bloom’s two-sigma gap. 

However, greater sophistication in one-on-one tutoring also pays off. Semb et al. 
(2000) reviewed several empirical studies of on-the-job training (OJT) and concluded that 
greater knowledge and use of tutorial techniques result in greater achievement and more 
efficient learning. These applications are found primarily in the military and industrial 
world, but they are, effectively, one-on-one tutoring. Including advanced tutorial techniques 
in our computer-based tutors may allow us to exceed Bloom’s two-sigma threshold. We 
may have just begun. 

F. PACE OF INSTRUCTION 

The possibility that simple approaches may, by themselves, do much to fill Bloom’s 
two-sigma gap is supported by considerations of pace: the speed with which students learn 
material and reach instructional objectives. Easily adjusted pacing is a capability claimed by 
even the most rudimentary CBI systems. 

Many teachers have been struck by the differences in the pace with which their stu- 
dents can learn. Consider, for instance, some findings on the time necessary for different 
students to reach the same instructional objectives: 

• Ratio of time needed by individual kindergarten students to build words from 
letters: 13 to 1 (Suppes, 1964) 

• Ratio of time needed by individual hearing-impaired and Native American stu- 
dents to reach mathematics objectives: 4 to 1 (Suppes, Fletcher, and Zanotti, 
1975) 

• Overall ratio of time needed by individual students to leam in grades K-8: 
5 to 1 (Gettinger, 1 984) 

• Ratio of time needed by undergraduates in a major research university to leam 
features of the LISt Processor (LISP) 4 programming language: 7 to 1 (Private 
communication, Corbett, 1998) 

As with the differences between one-on-one tutoring and classroom instruction, we 
may not be surprised to discover differences among students in the speed with which they 
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LISP is a language used for artificial intelligence (AI). AI pioneer John McCarthy formulated LISP in 
the late 1950s. 



are prepared to learn. However, the magnitudes of the differences may be much larger than 
we expect. As we might expect from Gettinger’s 1984 review, a typical K-8 classroom will 
have students who are prepared to learn in one day what it will take other students in the 
same classroom 5 days to learn. More homogeneous grouping of students based on their 
abilities does not seem to mitigate this difference. The students in Corbett’s 1998 university 
study are highly selected, averaging well above 1300 on their Standard Acceptance Tests 
(SATs); however, the differences in time they required to learn the fundamentals of a 
modestly exotic programming language remain large. 

Initially, students’ ability may cause the differences in the speed with which they 
reach given objectives, but prior knowledge quickly overtakes this effect as a determinant of 
pace (Tobias, 1982). Despite our efforts to sustain common levels of prior knowledge in 
classrooms by bringing every student to some minimal threshold of learning, we instead 
appear to increase the differences among students by about 1 year for every year they spend 
in elementary school (Heuston, 1997). For instance, the average spread of academic 
achievement in grade three is about 3 years. By grade six, it increases to about 6 years. We 
are, then, working hard to make the classroom teacher’s job more difficult. 

The challenge that this diversity presents to classroom teachers is daunting. How 
can they ensure that every student has enough time to reach given instructional objectives? 
At the same time, how can they allow students who are ready to surge ahead to do so? The 
answer, of course, despite heroic efforts to the contrary, is that they cannot. Most class- 
rooms contain many students who at one end of the spectrum are bored and at the other end 
are overwhelmed and lost. 

One-on-one tutoring allows us to alleviate this difficulty by adjusting the pace of 
instruction to the needs and abilities of individual students. We can proceed as rapidly or as 
slowly as needed. We can skip what individual students have mastered and concentrate on 
what they have not mastered. 

As with intensity or interactivity, we cannot make a direct cause-and-effect case for 
the contributions of individualized pace of instruction. However, with pace, as with inter- 
activity, we find that a large difference in instructional treatment is associated with a large 
difference in instructional outcome. It does not seem unreasonable to conclude that the abil- 
ity to adjust the pace of instruction may also account for some of the large differences that 
favor individual tutoring over classroom instruction. 
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Again, we might ask if CBI, like one-on-one tutoring, allows us to individualize the 
pace of instruction (with the pace defined by the amount of time it takes students to reach 
given instructional objectives). Research findings suggest that it does. If students who could 
move through instructional material more quickly are prevented from doing so in 
classrooms but are allowed to do so in CBI settings, we should find students reaching 
instructional objectives more quickly in CBI settings than in classroom settings. 

This finding arises repeatedly in reviews of instructional technology. Orlansky and 
String (1977) found that reductions in time to reach instructional objectives averaged about 
54 percent across 12 CBI evaluations used in military training. Fletcher (1991) found an 
average time reduction of 31 percent in 6 studies of interactive videodisc (IVD) instruction 
applied in higher education. Kulik (1994) and his colleagues found time reductions of 
34 percent in 17 studies of CBI used in higher education and 24 percent in 15 studies of 
CBI used in adult education. These reviews are effectively independent because they 
reviewed different sets of evaluation studies. From these reviews, it seems reasonable to 
expect time reductions of about 30 percent when students use CBI to reach a variety of 
given instructional objectives. 

It is not certain that these reductions result from the speed with which students pro- 
gress through fixed sets of items, from adjustments in content to take advantage of what 
students already know or have mastered, or from some combination of these. However, if 
we simply consider pace to be the rate with which students reach instructional objectives, it 
seems reasonable to conclude that CBI reduces time to learn, as does one-on-one tutoring, 
primarily by not holding back students who are ready to progress. 

There are three points to add to this discussion: 

• First, the self-pacing enabled by technology-based instruction does not simply 
allow students to skip through content as rapidly as they can. Instead, most 
and perhaps all technology-based instruction includes an executive agent. This 
executive agent might be called an instructional management system. It allows 
students to progress through content as rapidly as possible — but only after 
they demonstrate their readiness to do so. The instructional management sys- 
tem ensures instructional progress and quality in ways that books, as passive 
media, cannot. 

• Second, reducing instructional time by 30 percent is a fairly conservative tar- 
get. Commercial enterprises that develop technology-based instruction for the 
Department of Defense (DoD) regularly base their bids on the expectation that 
they can reduce instructional time by 50 percent, while holding instructional 
objectives constant. Noja (1991) has reported technology-based-instruction 
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time savings as high as 80 percent in training operators and maintenance tech- 
nicians for the Italian Air Force. 

• Third, time saved in learning is not a trivial matter. The Department of Defense 
spends about $4 billion a year on specialized skill training, which is the post- 
basic training needed to qualify people for the many technical jobs (e.g., 
wheeled vehicle mechanics, radar operators and technicians, medical 
technicians) needed to perform military operations. If DoD could reduce by 
30 percent the time to train 20 percent of the people undergoing specialized 
skill training, it would save over $250 million per year. If it could do so for 60 
percent of the people undergoing specialized skill training, it would save over 
$700 million per year. 

Assigning dollar values to the time that students spend in educational settings, espe- 
cially our K-12 classrooms, is more difficult. This difficulty may account for the paucity of 
results that can be found for time savings in K-12 education. However, time so spent or 
saved is not without cost and value. Aside from the obvious motivational issues of keeping 
students interested and involved in educational material, students who use their time well will 
profit themselves and any society that depends on their eventual competency and 
achievement. The time savings that technology-based instruction offers in K-12 education 
could be more significant and of greater value than the time savings obtained in post-educa- 
tion training. 

G. COST EFFECTIVENESS 

The issue for any educational decision-maker faced with a unyielding budget, an 
unpredictable revenue stream, and unending demands for expenditures that are both urgent 
and imperative is not limited to instructional effectiveness. The core of such decision- 
making is not just effectiveness but what must be given up to get it. Most often and most 
specifically, this consideration centers on costs and cost effectiveness. 

Is there evidence that technology applications in instruction are cost effective? 
Despite the decision-makers’ uncompromising need for such evidence, little evidence exists 
to aid their deliberations. This situation is especially prevalent among innovations, such as 
technology-based instruction, where researchers often seek to learn if an approach “works” 
or if it works better than existing practice but seldom to determine if it works well enough to 
justify its expense. The expense issue requires consideration of costs, cost models, and 
similar issues that researchers in instructional procedures and practices prefer to leave to 
others. An innovation’s cost effectiveness is rarely considered by anyone other than the 
decision-maker who will be pressured to adopt it. 



Of course, asking if an approach is cost effective oversimplifies the issue. Cost 
effectiveness is a relative term. We cannot meaningfully label some approach as cost effec- 
tive without specifying the alternatives with which this approach is being compared. Cost- 
effectiveness studies require the use of a single experimental paradigm — using comprehen- 
sive models of costs and effectiveness — to compare the alternatives under consideration. 
Typically, cost-effectiveness investigators either observe different levels of effectiveness 
achieved while holding costs constant or they observe the different costs required to reach 
fixed thresholds of effectiveness. 

Such comparisons in technology-based instruction are hard to find — even in indus- 
trial training where all decisions are a matter of profit and loss and in military training where 
allocations of resources may literally be a matter of life and death. Fletcher (1997) reported 
a limited cost-effectiveness argument for technology-based education. He presented 
empirical data gathered from earlier studies by Jamison et al. (1976), Levin, Glass, and 
Meister (1987), and Fletcher, Hawley, and Piele (1990) to compare the costs (adjusted for 
inflation) of different educational interventions to raise 5 th grade mathematics scores on a 
standard achievement test by one standard deviation. Providing daily 10-minute CBI ses- 
sions was found to be less expensive (and, hence, more cost effective) than peer tutoring, 
professional tutoring, decreasing class size from 35 to either 30 or 20, or increasing the 
length of the school day by 30 minutes. This finding suggests that a strong cost-effective- 
ness position for technology-based instruction is likely even at this early (relative to what 
may be coming) stage of development. Given that we are most likely in the horseless car- 
riage years of the third revolution in learning, these are promising results. More work of this 
sort is needed. 

H. INTELLIGENT TUTORING SYSTEMS 

If interactivity and individualization of pace can be achieved by standard approaches 
to CBI, is there any reason to pursue more exotic approaches? Specifically, is there any 
reason to develop what are called intelligent tutoring systems (e.g., Woolf and Regian, 
2000 )? 

“Intelligent tutoring systems” may be as intelligently, or unintelligently, designed 
as any others. They involve a capability that has been developing since the late 1960s (Car- 
bonell, 1970) but has only recently been expanded into general use. In this approach, an 
attempt is made to mimic directly the one-on-one dialogue that occurs in tutorial interac- 
tions. Carbonell was a computer scientist who focused on the underlying computation 
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capabilities needed to support this approach. He contrasted ad-hoc frame-oriented (AFO) 
approaches with information structure-oriented (ISO) approaches. Today, we might be more 
likely to discuss “knowledge representation” as the requisite capability, but, in either case, 
the software requirement is to represent human knowledge — knowledge of the subject 
matter, knowledge of the state of the student, and knowledge of teaching strategies. 

More important for those who wish to focus on instructional rather than computa- 
tional capabilities are the functionalities that distinguish intelligent tutoring systems from 
their predecessors. Despite current marketing efforts to describe any instructional system 
that uses technology as being an intelligent instructional system, clear differences exist 
between what has long been the objective of these systems and what has long been available 
in the state of the art. 

Two functionalities are critical and discriminating in an intelligent instructional sys- 
tem: 

• First, we expect to find an ability to generate computer presentations and 
responses in real time, on demand, and as needed or requested by learners. 

• Second, we expect to find an ability to support mixed initiative dialogue in 
which either the computer or the (human) student can generate, ask, and answer 
open-ended questions. 

Notably, instructional designers do not need to anticipate and pre-store these interactions. 
The motivation for funding the development of intelligent instructional systems in the early 
1970s was not to apply artificial intelligence (AI) or ISO techniques to CBI nor was it to 
mimic one-on-one tutorial dialogue. Instead, it was to reduce the costs of instructional 
materials’ preparation by developing capabilities to generate them on-line, in real time. 
Generative capabilities were intended to reduce the time and resources needed by the other 
approaches for anticipating and pre-specifying all possible student-computer interactions. 

Currently, intelligent tutoring systems are more sophisticated computationally and 
functionally than other more typical CBI systems, but they remain expensive to produce. 
These systems can, of course, adjust the pace, sequence, interactivity, style, difficulty, and so 
forth of instruction to the needs of individual learners, just as other approaches can. They 
can also make many of the adjustments to individual learners that human tutors can make. 
The cost to produce these systems will decrease as our techniques to develop them improve, 
but they may also be justified by increases in learner achievement. For instance, these 
systems show an increase in average effect size to 0.84 standard deviations (Fletcher, 1997), 
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which exceeds the average 0.42 standard deviations (e.g., Kulik, 1994) found for other CBI 
approaches. 

However, the main argument in favor of these intelligent tutoring systems is that 
they raise the bar for the ultimate effectiveness of technology-based instruction. Their 
unique generative and mixed initiative capabilities should eventually allow richer, more 
comprehensive, and more effective interactions to occur between students and the instruc- 
tional system. 

If Kurzweil (1999) is correct, we can expect a $1,000 unit of computing to equal the 
computational capability of the human brain by the year 2019 and exceed it thereafter. 
Computers may then become more effective in providing instruction than human tutors even 
if humans use all the techniques Graesser et al. (1995) found that they now neglect. We 
may not be implanting integrated circuits in our brains as Kurzweil suggests we might by 
2029; however, using computers to discover more than any human agent can about the 
unique potential of every individual and devising effective and individualized procedures to 
reach this potential seem to be appealing and realistic prospects. 

Whatever the case, the extensive tailoring of instruction to the needs of individual 
students that can be obtained by using generative, intelligent tutoring systems can only be 
expected to increase. Our current approaches may be reaching their limits. Intelligent 
tutoring systems may make available far greater instructional effectiveness and efficiencies 
than we can obtain from the approaches we are using now. 

L THE COLUMBUS EFFECT IN INSTRUCTIONAL TECHNOLOGY 

Prognostications aside, technological approaches to instruction may provide yet 
another example of what might be called the Columbus Effect. As readers will recall, 
Columbus sailed west intending to find India (and a lucrative spice route). Instead, he (re-) 
discovered what became a new world for Europeans. Such a result typifies technological 
progress. Seeking one thing based on familiar, common practice, we inevitably end up with 
something else — unforeseen and unexpected. Wireless telegraph produced something 
functionally quite different from the telegraph — namely, the radio. Similarly, efforts to make 
a carriage move without a horse produced automobiles, to say nothing of gas stations, 
motels, and the Santa Monica Freeway. In seeking affordable one-on-one tutoring through 
automation, we may end up with something no one now envisions. The metaphor based on 
current practice gets us started. The result may surprise us all. 
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As we begin with a vision of one-on-one tutoring made affordable through technol- 
ogy, our work may center on efforts to mimic the interactions that occur between human 
tutors and their students. We may be pursuing human-less tutoring (just as books, televi- 
sion, and other noninteractive media may be viewed human-less lecturing), but we are likely 
to end up with something quite different in function, appearance, and use. 

The advanced distributed learning (ADL) initiative currently being pursued jointly 
by DoD and the White House Office of Science and Technology Policy (OSTP) gives us a 
hint of what this different result may be. This vision is based on the expectation that most, if 
not all, human knowledge will become available as shareable, interoperable objects on the 
World Wide Web. The ADL initiative seeks to specify and develop these objects, but it 
does so because it envisions something that might be called a personal learning associate 
(PLA). 

Physically, a PLA will be a computer that is either carried or worn. In keeping with 
the suggestions of Kurzweil and others, it may even be implanted to provide direct brain- 
computer interaction, although this possibility seems more distant and needs more review 
and consideration. Functionally, a PLA will provide wireless connection to the Web or its 
successor in the global communication arena. A student of any age will use it for learning, 
and decision-makers (e.g., electronics technician, military tactician, or business planner) will 
use it to help solve practical problems. 

All users will interact with the PLA through spoken, natural language. It will provide 
a full range of display capabilities, including text, graphics, and photographs, as specified by 
the apocryphal Committee’s specifications. It will also provide animation, digital video of 
some sort augmented by a full range of high-fidelity sound, and, perhaps, tactile and haptic 
feedback. Olfaction remains under review. 

Most of the PLA’s physical capabilities are state of the art. Many of the software 
capabilities are also available or, like shareable courseware objects, soon will be (Fletcher 
and Dodds, 2000). Its instructional and decision-aiding functionalities remain longer term 
and more elusive. These functionalities call for the PLA to develop and then use compre- 
hensive, intimate, and accurate knowledge of the student/user to identify, collect, and inte- 
grate shareable instructional objects. This process will be accomplished on demand and in 
real time and will be precisely tailored to the individual’s needs, capabilities, interests, and 
cognitive style. If the intention is to help solve a problem, information that the PLA provides 
will not just be expert, but will also be delivered in a form that the individual can understand. 
If the intention is to establish a more permanent change in the individual’s cognitive ability 
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(i.e., to bring about learning), the PLA will do so efficiently and effectively in ways far 
superior to those that we now imagine. 

Obviously, we have a way to go to realize this vision. What form it eventually takes, 
how it is used, what infrastructure it engenders, and what impact it has on our lives remain to 
be seen, but its key capabilities may arise from the intelligent instructional systems we are 
now learning to build. The Columbus Effect will kick in sooner or later, but beginning with 
a guiding metaphor based on individualized, tutorial instruction and “mentoring” seems a 
good way to advance. The goal of learning to do something that is within our reach but 
outside our grasp has long been a stimulus for human progress. 

J. IMPACT ON RESEARCH AND THEORY 

Reviews by Krendl and Lieberman (1988) and Schater and Fagnano (1999) con- 
tinue to echo earlier recommendations by Suppes (1964) and others to apply advances in 
cognitive and learning theory to the development of technology-based instruction. Such 
efforts will improve the quality of instruction delivered. More importantly, these efforts will 
provide feedback to theories of cognition and learning to indicate where they are right, 
where they might use some improvement, and where they have left serious gaps that need to 
be filled. This is the traditional interplay of theory and empirical research that has served 
other areas of systematic investigation so well. Technology-based instruction, with its pre- 
cise control over stimulus inputs and equally precise measurement of response outputs, 
should play a unique role in completing the feedback loop between theory and empirical 
research. 

Such feedback will produce significant advances not just in instruction, but also in 
related areas. Instructional applications that use technology test the notions — or theo- 
ries — of human cognition, learning, and instruction embodied in them in at least two ways: 

• First, the ability to put a proposed notion, model, or theory into a computer 
algorithm is a significant demonstration. If a notion cannot be captured by an 
algorithm, it may not be testable. If it is not testable, it is not worth serious 
consideration. 

• Second, an instructional application is an instantiation that tests the correctness 
of a notion, model, or theory. To the extent that an application achieves its 
goals, the notion(s), model(s), or theory of cognition, learning, and instruction 
on which it is based can be viewed as correct. However, tests of instructional 
applications seldom yield simple answers. The fine-grain data that technol- 
ogy-based instruction makes available will increase the richness of the feedback 
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we receive. Few theories will be shown to be perfectly and thoroughly correct. 
The more complete diagnostic information concerning where they are correct, 
where they are not correct, and what they lack is critical. Detailed and specific 
information of this sort produces significant advances in other fields, and we 
should expect nothing less from our instructional applications. 

K. THE ENGINEERING OF INSTRUCTION 

Beyond issues of feedback for theory, our third revolution in learning may effect a 
shift in instruction from art to engineering. We tend to view teaching as an intensely 
“human” activity, something that at best can only be accomplished by human teachers 
interacting with human students. Master teachers do exist, and many of us have benefited 
from the attentions of at least one teacher who, if not a master, was at least able to impress 
on us the value, benefits, and pleasure of scholarship. However, such occasions appear to be 
more the exception than the rule. Our perceptions of teaching as a warmly experienced, 
human activity vary with much that we experience in real classrooms, where each student is 
one of many waiting for that portion of instruction that addresses his or her individual 
needs. 

To a great extent, successful instruction is a matter of design — the creation of an 
environment to maximize the probability that learning will occur and that every student will 
achieve specified instructional objectives. Mostly, we seek and test for accuracy of know- 
ledge. However, we may also seek other objectives such as speed of response, retention, 
insight, and transfer of knowledge as well as continued interest in and respect for the subject 
matter. These goals may be compatible to some extent, but, at some point, they diverge and 
require different approaches that compete with each other for classroom resources of human 
energy, funding, and, especially, time. 

The design of such an environment may be viewed as an art — a highly personal, hit 
or miss affair. To be fair to our students and productive in our instruction, we need to 
establish a science of design that allows these many instructional objectives to be accom- 
plished by many different hands. In short, we need an engineering of instruction in which 
specific designs reliably yield specific instructional outcomes. 

The notion of instruction as engineering may be unpopular. Raymond Fox (1994) 
has noted that “one of the more difficult problems in dealing with improvement in public 
education is to replace the notion of teaching as an art form with that of instruction delivery 
as a systems science” (p. 2). Without technology, precise engineering to ensure that every 
student achieves a wide range of objectives would be out of the question. Different human 



teachers, specifically because they are human, must approach their students differently and 
succeed in different ways with different students. One-on-one tutoring can accomplish this 
goal to some degree, but the precision, “patience,” and consistency of technology may sur- 
pass what we can accomplish with human tutorials. A technology-based approach may 
provide the individual, precisely tailored attention that each student needs to achieve fully his 
or her potential. 

L. IMPACT ON OUR CURRICULAR GOALS 
In 1960, T.F. Gilbert wrote: 

If you don’t have a gadget called a “teaching machine,” don’t get one. 

Don’t buy one; don’t borrow one; don’t steal one. If you have such a 
gadget, get rid of it. Don’t give it away, for someone else might use it. This 
is the most practical rule, based on empirical facts from considerable 
observation. If you begin with a device of any kind, you will try to develop 
the teaching program to fit that device [p.478 (the italics are Gilbert’s)]. 

Gilbert may be right and wrong. He is certainly correct in suggesting that instruc- 
tional designers and developers who use a “teaching machine” will try to fit the teaching 
program to it. The new functionalities that such a device makes available motivate its use in 
the first place. One can imagine students long ago poring over clay tablets or papyrus rolls 
once their teachers learned how to design teaching programs to take advantage of written 
language. The same can be said for printed books, which, because they were more 
accessible and less expensive than papyrus rolls or codices, allowed teachers different 
assumptions in the development of their teaching programs. The same is doubtlessly true 
for our third instructional revolution, which applies interactive, computer-based technology 
to the problems and processes of instruction. 

It is less certain whether such adaptations (e.g., fitting the teaching program to the 
“device”) are significant evils to be avoided at all costs. Certainly, if technology causes us 
to remove or de-emphasize essential elements of our teaching programs, it will diminish 
their effectiveness. However, if technology is properly applied, it will improve, if not 
revolutionize, the effectiveness and efficiency of our teaching programs. Researchers, 
developers, and instructors — not the technology itself — must ensure that it does. 

In either case, the application of technology will change what and how we teach. 
Technology will raise the bar for our curricular aspirations. Tutorial simulations will afford 
our students experiences, access to exotic (i.e., expensive and unavailable) devices, and 
immersion in collaborative problem solving that we could not provide in any other way. 





Intelligent tutoring capabilities will permit tutorial interactions or simple conversations with 
experts and expertise that would otherwise be out of the question. Because these interac- 
tions will be geared to each student’s level of ability and prior knowledge, they will produce 
levels of understanding that would otherwise be unattainable. Sooner or later, we will be 
forced by necessity, public pressure, or our own professional integrity to adapt our teaching 
programs to the new functionalities that technology makes available. The most important 
consequence of the third revolution in instruction may not be that it improves efficiency and 
effectiveness of what we do now, but that it changes what it is that we choose to do. 

M. SUMMARY 

More discussion needs to take place about the third revolution in learning wrought 
by technology, but that discussion must await wiser, better-informed commentary. However, 
with modest certainty, a few statements can be made. This revolution will: 

• Make the functionalities of individualized tutoring widely accessible and 
affordable 

• Permit interactive, individualized learning to take place anytime, anywhere 

• (Eventually) bring about profound changes in our educational institutions and 
the roles and responsibilities of people (teachers, students, and administrators) 
in these institutions 

• Help bring into being a nation of life-long learners who are well prepared to 
meet the challenges of the new century and thrive in the global marketplace 

• Lead to capabilities, uses, and functionalities of which we are now only dimly, if 
at all, aware (thanks to the Columbus effect) 

• Produce radical change in the practice and processes of instruction. 

Basically, people learn just one way, and this way mostly likely requires growth or 
chemical changes in the synapses. The third revolution in learning will not change learning 
at this level any more than writing or books did. However, substantially increasing the 
probability that such fundamental changes will occur across all manner of humans in all 
manner of environments does seem to qualify as a radical change — one that warrants the 
term “revolution.” Because this revolution will increase the tempo with which learning 
occurs, it might be called a revolution in learning, the magnitude of which we have only seen 
twice before in human history. 
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